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Philip Abrami and his colleagues at Concordia University have done an admirable job of 
over viewing the complex state of research and related perceptions of e-learning in 
Canada in the early years of the 21 st Century. The work of these scholars at quite 
effectively aggregating and synthesizing is commendable given the emergent, regionally 
disparate, complex and most critically, under researched domain that they have reviewed. 
Unfortunately, what stands out most readily for me (and echoed by the authors) is the 
poverty of data, sound research, and systematic analysis of the considerable social, 
economic and educational impact that has accompanied the rapid integration of e-learning 
technologies into both formal and informal learning institutions and organizations. This 
dearth of data is especially ironical and perplexing given the positive views of almost all 
respondents (from educators to policy makers, from students to the general public) 
identified in the review. Canadians seem to very clearly see e-learning as a critically 
important component of life and learning. However that interest has not evolved into an 
active e-learning research and development and knowledge dissemination program 

The report begins with a definition of e-learning—one developed by the Canadian Council 
for Learning (CCL) who are sponsors of the study. Their definition seems to adequately 
define the diverse set of practices and technologies in actual use. "E-learning is the 
development of knowledge and skills through the use of information and communication 
technologies (ICTs) to support interactions for learning—interactions with content, with 
learning activities and tools, and with other people" (Rossiter, 2002). I was especially 
pleased to note the focus on interaction—with both humans, activities and content that is 
included in the definition. In the tradition of John Dewey, Vygotsky and Paiget, I have 
argued and tried to further delineate the various types of interactions and noted its effect 
on learning motivation, achievement and persistence in educational contexts (Anderson, 
2003) including those created on campus and in online environments. I also appreciated 
the acknowledgement by the authors and by CCL of the convergence of tools and practice 
between formerly distinct modes of distance and campus based education into models in 
which e-learning plays significant, if not dominant importance. I doubt that I could craft a 
better definition, but I note that the breadth and extensiveness of the definition, allows 
inclusion of very diverse forms of instruction and learning ranging from the use of a 
computer in a classroom for drill and practice to completion of complete graduate degrees 




online, with no face-to-face meetings. Both of these are examples of e-learning but it 
stretches my mind past the point of credibility to think how "what works" in one context 
applies to "what works" in the other. 

The review attempts to answer a quixotic question that plagues every intervention in 
formal and informal learning. What practices and procedures relating to e-learning use are 
most effective in promoting learning? Given the pragmatic nature of many Canadian 
educators, politicians and parents, this is an important question. It follows that a thorough 
reading of a broad range of research, professional and popular writing should provide both 
specific and broad policy guidelines, best practices and specific activities that work 
effectively with e-learning technologies and techniques. Unfortunately, the authors admit 
in the executive summary that "it seems that studies that can help us understand 'what 
works' in e-learning are underrepresented in Canadian research literature." Before 
reviewing the attributes of the study it is worth a short digression into the nature of the 
research question. 

"What works" are American code words for a particular type of epistemological worldview, 
with strong roots in the empirical and positivist traditions of scientific research. Of course, 
many forms of knowledge can aide teachers, learners and policymakers in determining 
what works in their context; however, in recent years these words have been high-jacked 
to mean a particular type of evidence—thereby excluding other types. A quick visit to the 
American "what works" web site (http://www.whatworks.ed.gov/) reveals that evidence is 
only considered useful in determining "what works" if it clearly exhibits "a causal effect on 
student outcomes. In research that focuses on causal effects, experimental research 
designs provide the strongest evidence of what works, followed by certain types of quasi- 
experimental research designs." I have two major concerns with this approach. The first is 
the misappropriation of the terms "what works" to exclude other forms of knowledge that 
may provide equal or even greater insight into determining the effect or impact of an 
intervention. For example the contextual influences of an individual or group's state of 
health, motivation, access, time availability and other internal or external conditions may 
have tremendous effect on "what works", yet if the subjects were not subjected to control 
(an almost impossible task in real world contexts), these large contextual variables are 
often ignored. Second, the absolute sense of "what works" implies that the particular 
intervention will work regardless of the context in which it is used. Education and learning 
happen in wide variety of contexts determined by a very wide range of cultural, economic, 
individual, discipline and other variables ( Figueiredo , 2005) . Neither lifelong learning nor 
formal educational schooling take place in controlled laboratories with identical subjects; 
to suggest that there are interventions that "work" regardless of the state of these 
variables, is to disregard the emergent complexity of any learning context. 

Fortunately, the authors are veterans of this ongoing "paradigm war" and so have wisely 
included reviews of qualitative, narrative and opinion pieces. They have made a valiant 
and productive attempt to shoehorn these diverse artefacts into an understandable whole, 
by applying a series of tags and structured vocabularies that allow broader trends, effects 



and conclusions to be extracted from the whole than from any individual article or study. 
Of course the more personal, naturalistic and qualitative the article is the harder it is to 
stuff into a set of common criteria—and especially criteria that permit mathematical 
manipulation to determine combined effect sizes. 

The authors follow two methodological approaches. The first allows use of the qualitative 
pieces by having coders classify each study bilaterally among a series of broad 
classifications. For example, was e-learning use positively, negatively or neutrally 
associated with a diverse set of educational outcomes including motivation, achievement , 
interactivity, attrition, and so on. The means of these classifications were calculated, 
averaged to produce a general effect of e-learning and then the correlations of the studies 
were cal cul ated based upon the media used, sector, locations, designs and technology 
types and other variables asso dated with each study. Many of the variables did not 
produce significant differences on the overall effect of e-l earning, but some did. As 
expected, studies that noted infrastructure needs, professional development, and were 
dependent upon e-learning (rather than being an optional add- on) had significant positive 
effect on the combined e-learni ng outcome. I was also pleased to note that use of e- 
learning for communication applications and for student (as opposed to teacher) use of the 
technol ogies were associated with higher positive effect. 

The second more classic multi-source technique is a meta-analysis in which an overall 
effect size is calculated by combining the effect size of numerous, similar studies. One can 
sense the frustration of the authors as 2,042 documents scanned are reduced to 726 of 
real value to the question from a Canadian context, then reduced to 152 with empirical 
data and finally to only 17 studies of an experimental or quasi-experimental design that 
provided an effect size and sample size n's—critical factors in allowing entry into a 
traditional meta-analysis. As expected, even here heterogeneity among studies was 
significant, which illustrates that the variation amongst contexts and applications of e- 
learning is very high. As a result, one can not place a great deal of faith in the very small 
positive effect size that was calculated from the data from these 17 studies. As is now 
traditional in all educational meta-analysis studies that I have seen, the authors complain 
bitterly about the absence of quantity of useable studies and a host of methodological 
deficiencies. What is often left un-discussed is the validity of combining studies from very 
disparate contexts; each with only the common characteristic that e-learning is used in 
some way. 

Results 

The report synthesizes five consensus positions amongst targeted groups of researchers, 
reviewers, policy makers and practitioners. The first is the focus on individual readiness, 
which underlines the need for individuals (and I would argue institutions) to have access, 
skill and attitude conducive to availing themselves of the affordances of this new net- 
enabled context. This conclusion underlines the need for e-learning to meet the high 
expectations of Net generation youngsters as well as for those who are immigrants to this 
very foreign, confusing and sometimes frightening new context for learning. The second 



conclusion echoes decades of instructional design research and underscores that it isn't 
the tool, but how the tool is conceived of and used that makes the difference for learning. 
E-learning is much more than "putting the lectures online" and requires both pedagogical 
and epistemological adaptation to maximize its effectiveness. From this need for change 
follows the third conclusion noting that teachers (and others involved in the education 
business) require focused professional development. The fourth consensus notes the 
expansion of communicative options afforded by the Net and the potential for increases in 
collaborative and cooperative work leading (hopefully) to development of higher-order 
thinking skills. While this is certainly true and the mantra of E-learning 2.0, the second 
consensus reminds us that creating opportunity for interaction is no guarantee of its 
incidence. The fifth consensus points to impressive increases in access, with a special nod 
to that provided to disabled learners. Although I certainly concur with the tremendous 
increase in access afforded through this technology, I would not limit this to disabled 
students (after all, we are all disabled to some degree). But most importantly, the net 
provides students with freedom in multiple dimensions including control of space and pace, 
content, relationship, media and in some cases to cost (Paulsen, 1993) . In sum, the 
consensus conclusions accurately reflect the major challenges and the major affordances 
of the Net and it is reassuring to see these reflected in the Canadian data reported in this 
study. 

The National/International context 

In the introduction, the authors claim that in e-learning "there is sufficient optimism for 
technology's positive impact that governments have established committees, formed task 
forces, and dedicated substantial funds to the delivery or enhancement of technology- 
based instruction" (p. 10). I argue that in fact Canada very significantly trails behind 
comparative countries in the size and quality of investment in e-learning policy making, 
implementation and especially in systematic research. For example, the governments of 
Australia, (http://www.flexiblelearning.net.au); Ireland (www.ncirl.ie/downloads/research 
_and_innovation/SFI_reportFINAL.pdf) and the United Kingdom (www.dfes. 
gov.uk/publications/e-strategy/), amongst many others, have articulated strategies by 
which they support and accelerate the use of e-learning throughout their national 
jurisdictions . This type of national or pan-Canadian (to be politically correct) concern and 
investment in e-learning and leadership is sorely missing in Canada. Ironically, the 
reference the authors use to defend this claim of support is from the Council of Ministers of 
Education, Canada. In fact, this reference is a whining press release by the provincial 
education ministers that urges the federal government to invest in connectivity for e- 
learning. But there was no evidence then, nor is there any today, that demonstrates these 
same provincial Ministers' willingness to invest funds in this provincially mandated activity! 

Methodology 

One of the impressive components of the report, and one that we have become 
accustomed to in work from the Centre for the Study of Learning and Performance at 
Concordia University, is the transparency and thus potential replication of the study 



methodology. Details with regard to choice of documents to be included, the Argument 
Catalogue used for analysis and the methods of that analysis are clear and forthcoming. 
This is especially important when dealing with content that is as challenging to categorize 
and conceptually deal with as the diversity of forms and functions of e-learning research 
and practice. The work also provides a foundation, model and motivation for further 
systemic review. 

I particularly liked the brief summations of the non-research publications in which the 
authors' attempt to summarize prevailing public, policy maker and practitioner attitudes 
towards e-learning are presented. These synopses reveal critical attitudes that both propel 
and confound efforts by participants to develop and exploit e-learning technology and 
techniques. I also commend the authors for accurately exposing some of the concerns of 
critics (Oliver & Conole, 2003) that are similar to my own concerns noted earlier with 
regard to methodological hegemony. Indeed, as Cantoni and Rega (2004) noted, there is 
no consensus amongst researchers on means, methods or epistemology that should guide 
e-learning research—thus necessitating an eclectic approach that attempts to winnow the 
best from both complementary and contradictory sources of evidence. Canadian 
researchers have, for the most part, stood on the side (not even acting as UN 
Peacekeepers!) as paradigm wars rage in the USA and other countries. Perhaps this 
reflects our national reluctance to converge our thinking on any single solution and we are 
destined to creating a low-energy, multi-cultural and multilinguistic e-learning research 
agenda in this, as in many other avenues of Canadian life. 

I was also not surprised to find that 20% of the articles in the sample studied by the 
reviewers looked at technology adoption and integration. This very significant focus, along 
with a slightly smaller focus on professional development, indicates the critical change and 
innovation management issues that are integral to the study of e-learning in real contexts. 
E-learning is a disruptive technology (Anderson, 1999; Archer, Garrison, & Anderson, 1999; 
Norman, 1999) and one that poses considerable threat to the way in which our mostly 
conservative institutions (universities, colleges and schools) operate. The large number of 
articles focused on these change processes is heartening and shows that researchers are 
at least aware of the problems associated with adoption of e-learning technologies. In a 
listing of eight requirements for adoption of technology-based innovation in education, Ely 
(1999) lists a " Dissatisfaction with the status quo" as the first necessary condition. In an 
era of growing student enrolments, teacher shortages and increasing emphasis on 
commercialization ( Bok , 2003) , there seems little or at best sporadic effort at reform and 
innovation needed to propel long term rethinking of teaching and learning (Pocklington & 
Tupper, 2002) . This is especially sad given the opportunity and demand for learning 
created by the affordances of the information rich and connected society ( Downes, 2005; 
Siemens, 2005; Willinsky, 2005) . 

The report does a nice job of categorizing the evidence based upon the key target 
audiences of the Council for Learning (adult learning, K-12, and so on) The overall impact 
score of e-learning is positive for all sectors, but it is interesting to note that over half the 



studies were situated in post secondary contexts. This is not because Canadians spend 
more money or time in postsecondary as opposed to K-12 or adult education/training but 
rather because the vast majority of the small number of active researchers in e-learning 
are employed at post secondary institutions and thus the convenience samples obtained in 
their own institutions are most studied. A more strategic research agenda would focus e- 
learning research in areas where we currently have the most expense or in which we have 
a sense of greatest potential benefit. My own bet is that continuing lifelong and 
professional education offers the greatest potential yet totalling adult, health and the 'not 
specified' yields only 18% of the e-learning studies. 

Conclusion 

I was somewhat surprised to have the author's conclude with a worrying note about the 
future of e-learning "given the magnitude of the investment required to develop it, sustain 
it and cause it to grow" (p. 45). 2005 data from Statistics Canada showing that 67.9% of 
Canadians access the internet regularly 

(http://www40.statcan.ca/l01/cst01/comml4a.htm) and 2004 data from the US showing 
that the number of learners enrolled in online courses is growing by over 18% per year 
(Sloan, 2005), indicate that e-learning is very likely here to stay. What is in question is 
Canada's role in designing, distributing, researching and leading in this domain. The 
distributed and dispersed nature of our major education, commercial and social institutions 
and population woul d make one think that Canadians and their governments would place 
high priority on being not just consumers but world class leaders and producers of e- 
learning. 

The report concludes with a now traditional call for greater support and more strategic 
funding on the most important components of the educational system. I could not agree 
with more enthusiasm with the key recommendations that conclude the report and these 
are worth repeating here: 

• Need for money spent on research development and evaluation to more closely align with the 
huge amount spent on education provision. 

• The importance of adopting longitudinal, multi-contextual and appropriately sized research as is 
common practice in health study 

• The need to focus on professional development and support of educators (Ely, 1999) 

• The need to focus on instructional design and the many contextual variables that effect learning 
in the diverse learning contexts of formal learning 

• The need for research that includes the design and development of new models and techniques 
of e-learning. 

• As the authors note, money alone (and especially that thrown exclusively at hardware 
acquisition) does not solve all problems, but the lack of investment in this strategic area is 
jeopardizing our capacity to meet the lifelong learning needs of Canadians. We should and we 
can do more. The field is far too new, development opportunities are too great and the need for 
involved commitment of researchers, e-learning businesses and practitioners is too pressing, to 
rely on products and processes developed by non-Canadians. This report challenges and inspires 
all of us from policy makers to e-learning consumers to insure that Canadians are able to take 
advantage of this most important educational development since the printed text. 

References 



Anderson, T. (1999). Using disruptive technologies in the Universities: Confessions of an 
Agent Provocateur. In Ed-Media 1999 World Conference on Educational Multimedia, 
Hypermedia and Telecommunications: AACE. 

Anderson, T. (2003). Modes of interaction in distance education: Recent developments and 
research questions. In M. Moore &.W. Anderson (Eds.), Handbook of Distance Education. 
(pp. 129-144). Mahwah, NJ: Erlbaum. 

Archer, W., Garrison, D. R., & Anderson, T. (1999). Adopting disruptive technologies in 
traditional universities: Continuing education as an incubator for innovation. Canadian 
Journal for University Continuing Education, 25(1), 13-30. Retrieved September 2006 
from http://www.extension.usask.ca/CJUCE/enq/Ab251.html 

Bok, D. (2003). Universities in the marketplace: The commercialization of higher 
education. Princeton, NJ: Princeton University Press. 

Cantoni, L., & Rega, I. (2004). Looking for fixed stars in the elearning community: A 
research on referenced lit. in SITE proceeding book from 1994-2001. Proceedings of World 
Conference on Educational Multimedia, Hypermedia and Telecommunications (Vol. 2004, 
pp. 4697-4704). Retrieved September 2006 
from www.newmine.org/newmine cantoni alii EDMEDIA04.pdf 

Downes, S. (2005). E-Learning 2.0. ELearn Magazine, Retrieved Dec. 2005 
from http://elearnmaq.orq/subpaqe.cfm?section = articles&article = 29-l 

Ely, D. P. (1999). Conditions that facilitate the implementation of educational technology 
innovations. Educational Technology, (39), 23-37. 

Figueiredo, A. (2005). Learning contexts: A blueprint for research. Interactive Educational 
Multimedia, (11), 127-139. Retrieved July 2006 

from http://www.ub.es/multimedia/iem/down/cll/Learninq Contexts.pdf 

Norman, D. (1998). T he invisible computer: Why good products can fail, the personal 
computer is so complex and information appliances are the solution. Cambridge, MA: MIT 
Press 

Oliver, M., &Conole, G. (2003). Evidence-based practice and e-learning in higher 
education: Can we and should we? Research Papers in Education, 18 (4), 385-397. 

Paulsen, M. (1993). The hexagon of cooperative freedom: A distance education theory 
attuned to computer conferencing. DEOS, 3(2) Retrieved May 28, 2004 
from http://www. nettskolen.com/forskninq/21/hexaqon.htm I 

Rossiter, J. (2002). An e-learning vision: Towards a Pan-Canadian strategy and action 
plan. (CANARIE Discussion paper). Ottawa, ON: CANARIE Inc. Retrieved August 2005, 
from http://www.canarie.ca/fundinq/elearninq/elea rninqvision.pdf 



Pocklington, T., &Tupper, A. (2002). No place to learn: Why universities aren't working. 
Vancouver: UBC Press. 

Siemens, G. (2005). A learning theory for the digital age. Instructional Technology and 
Distance Education , 2(1), 3-10. Retrieved October 2005 
from http://www.elearnspace.orq/Articles/connectivism.htm 

Sloan Foundation (2005) Growing by degrees: Online education in the United States , 
2005 .Retrieved October 2006 from http://www.sloan- 
c.orq/publications/survev/pdf/qrowinq by deqrees.p df 

Willinsky, J. (2005). The unacknowledged convergence of op en source, open access and 
open science. First Monday, 10(8) Retrieved August 2006 
from www.firstmondav.org/issues/issue 10 8/willinsky/ 


© Canadian Journal of Learning and Technology 

ISSN: 1499-6685 







